knowledge: kubelet resource-metrics, PID1 log flush, date %3N detection#4
Open
dch0202-rsquare wants to merge 2 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Knowledge flush — 3 insight(s)
Drained 3 queued
★ Insightcandidates. Each was independently researched andverified against primary sources before ingest. Two new pages created, one
existing page extended. No auto-merge — this PR is for review.
Cross-Check: each of the 3 candidates was verified by an independent research
subagent against primary sources (chart values.yaml / k8s docs / kubernetes-mixin;
Greg's Wiki + pipe(7) + Docker/bash manuals; GNU + FreeBSD date + POSIX strftime).
All three mechanisms VERIFIED; corrections folded in (mixin also pins
job=cadvisornot justimage!=""; process-sub$!is bash 4.4+ not 5.1) andfield-observed specifics (OrbStack root cause, literal
3N) labeled as such.Verified best-practice
1. Empty cAdvisor
container_*metrics on a lightweight kubelet → scrape/metrics/resourcekubelet targets are UP; fix by enabling
kubelet.serviceMonitor.resource: trueand overriding
resourcePath: /metrics/resource; mixin dashboards hide theresource series via
image!="".resourcedefault false, andresourcePathdefault is the stale/metrics/resource/v1alpha1: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml/metrics/resourceis the current STABLE endpoint; series carrynamespace/pod/containeronly (noimage): https://kubernetes.io/docs/reference/instrumentation/metrics/ , https://kubernetes.io/docs/concepts/cluster-administration/system-metrics/v1alpha1path was removed (404 on kubelet ≥1.20): Refactor kubelet resource metrics kubernetes/kubernetes#86282image!=""andjob="cadvisor"/metrics_path="/metrics/cadvisor": https://github.com/kubernetes-monitoring/kubernetes-mixinimage!=""is not the only reason the resource series are excluded — the mixin also pinsjob=cadvisor/metrics_path=/metrics/cadvisor, so a custom panel is required regardless (the page states this, not "just drop the image filter"). The OrbStack-specific root cause (cAdvisor emitting onlymachine_*) is field-observed, not documented upstream — labeled as such in the page and log.2. Container PID 1 entrypoint loses piped logs on exit → wait for tee in an EXIT trap
exec > >(tee -a file)in a PID 1 bash entrypoint loses log output oncontainer exit; capture
TEE_PID=$!andwaitit in an EXIT trap after closingfds; add
trap 'exit 143' TERM.$!to its PID so you canwait: https://mywiki.wooledge.org/ProcessSubstitution$!is bash 4.4+ (2016), NOT 5.1 as the raw candidate implied — the page states 4.4+ and gives a named-FIFO fallback for older bash. EOF caveat (all write ends must close) captured as an edge case.3. Feature-detect
date +%3Nby 3-digit output, not by%Nsupportdateby "does%Nwork" is wrong because modernmacOS supports
%N; check thatdate +%3Noutputs three digits.%Nnanoseconds + numeric field-width form: https://www.gnu.org/software/coreutils/manual/html_node/Options-for-date.htmldate—%Nadded in FreeBSD 14.1, GNU-compatible on modern macOS; no%3Nwidth form documented: https://man.freebsd.org/cgi/man.cgi?query=date%N, making it an even worse discriminator than the candidate claimed. Two specifics are field-observed: the exact%3N→ literal3Noutput (consistent with POSIX "undefined", not documented as3N), and GNU%3N= "first 3 digits" (width mechanism documented; the truncation semantics universally relied on but not spelled out). The page frames3Nas field-observed.Existing-layer check
infrastructure/observability/pages (logs-metrics-signals,alerting) and allinfrastructure/containers/pages (image-builds,resource-limits-and-probes), plusplatforms/tools/bsd-vs-gnu-cli,platforms/shells/portable-shell-scripts,platforms/environment/timezone-and-locale.logs-metrics-signalscovers what/how to instrument(signal choice, cardinality, golden signals) — a different trigger from "a
specific series is empty despite a healthy scrape." Merging would blur that
page's focus, so created a new page and cross-linked both ways
(
related:). No conflict.portability, so it does not belong in
platforms/shells/portable-shell-scripts(which already covers process substitution/
set -e/EXIT traps for theportability case). Created a new page under
infrastructure/containers(owns the PID 1 lifecycle artifact) and added a
related:link both ways tothe shell page. No conflict.
date %3N): directly overlaps the existingdateguidance inbsd-vs-gnu-cli→ merged (one Do-table row + one Instead-of row + twosources +
last_verifiedbump). No new page. No conflict with the existing"Relative date" row (distinct sub-case).
Routing decision
infrastructure/observability/kubelet-resource-metrics.md(new page)infrastructure/index.mdobservability table;related:↔logs-metrics-signalsinfrastructure/containers/pid1-entrypoint-log-flush.md(new page)infrastructure/index.mdcontainers table;related:↔portable-shell-scripts, →resource-limits-and-probes,logs-metrics-signalsdate +%3Ndetectionplatforms/tools/bsd-vs-gnu-cli.md(existing)last_verified→ 2026-07-15No new category was needed — both new pages fit existing categories
(
infrastructure/observability,infrastructure/containers).log.mdhas threeappended entries (2× ingest, 1× revise).